Parameter Estimation
In principle, a full Bayesian treatment requires us to specify prior distributions over the model parameters $\theta = \{ \{\pi_k\}, \{\mu_k\}, \Sigma \}$, and then apply Bayes rule to obtain the corresponding posterior distributions. While this is certainly possible, the mathematics quickly becomes bewildering. Therefore, we opt for maximum likelihood estimation of the parameters as a more practical alternative.
Show that the discrimination boundaries for the posterior class probabilities in Eq. (4) evaluates to a line (or hyperplane).
The features are stored in two matrices, corresponding to the two classes:
🎯 Key concept
For the GCM with equal covariances, the posterior class probability is a softmax function,
$$ p(\mathcal{C}_k|x,\theta ) \propto \exp\{\beta_k^T x + \gamma_k\}$$
where $\beta _k= \Sigma^{-1} \mu_k$ and $\gamma_k=- \frac{1}{2} \mu_k^T \Sigma^{-1} \mu_k + \log \pi_k$.
Exercises
🎯 Key concept
A common generative model for classification is the Gaussian–Categorical model:
$$p(x_n,\mathcal{C}_k|\,\theta) = \pi_k \cdot \mathcal{N}(x_n|\mu_k,\Sigma) \,,$$
leading to
$$p(x_n,y_n|\,\theta) = \prod_{k=1}^K \big( \pi_k \cdot \mathcal{N}(x|\mu_k,\Sigma)\big)^{y_{nk}} $$
The ML for multinomial class prior (we've done this before!)
$$\begin{align*} \hat \pi_k = \frac{m_k}{N} \end{align*}$$
Now group the data into separate classes and do MVG ML estimation for class-conditional parameters (we've done this before as well):
$$\begin{align*} \hat \mu_k &= \frac{ \sum_n y_{nk} x_n} { \sum_n y_{nk} } = \frac{1}{m_k} \sum_n y_{nk} x_n \\ \hat \Sigma &= \frac{1}{N} \sum_{n,k} y_{nk} (x_n-\hat \mu_k)(x_n-\hat \mu_k)^T \\ &= \sum_k \hat \pi_k \cdot \underbrace{ \left( \frac{1}{m_k} \sum_{n} y_{nk} (x_n-\hat \mu_k)(x_n-\hat \mu_k)^T \right) }_{ \text{class-cond. variance} } \\ &= \sum_k \hat \pi_k \cdot \hat \Sigma_k \end{align*}$$
where $\hat \pi_k$, $\hat{\mu}_k$ and $\hat{\Sigma}_k$ are the sample proportion, sample mean and sample variance for the $k$th class, respectively.
In the scatter plot, the two features are represented along the two $x$-coordinates, while the fruit label $y \in \{\text{apple}, \text{peach}\}$ is encoded by the marker style.
Exercise: Evaluate log-likelihood
Click for the solution
(a) We choose $C_1$ if $p(C_1|x)/p(C_2|x) > 1$. This condition can be worked out as
$$\frac{p(C_1|x)}{p(C_2|x)} = \frac{p(x|C_1)p(C_1)}{p(x|C_2)p(C_2)} = \frac{10 \times 0.6}{200(x-1)\times 0.4}>1 $$
which evaluates to choosing
$$\mathrm{Decision} = \begin{cases} C_1 & \text{ if $1.0\leq x < 1.075$} \\ C_2 & \text{ if $1.075 \leq x \leq 1.1$ } \end{cases}$$
The probability that $x$ falls outside the interval $[1.0,1.1]$ is zero.
(b) The total probability of error is
$$p(\text{error})=\int_x p(\text{error}|x)p(x) \mathrm{d}{x} \,.$$
We can work this out as
$$\begin{align*} p(\text{error}) &= \int_x p(\text{error}|x)p(x)\mathrm{d}{x}\\ &= \int_{1.0}^{1.075} p(C_2|x)p(x) \mathrm{d}{x} + \int_{1.075}^{1.1} p(C_1|x)p(x) \mathrm{d}{x}\\ &= \int_{1.0}^{1.075} p(x|C_2)p(C_2) \mathrm{d}{x} + \int_{1.075}^{1.1} p(x|C_1)p(C_1) \mathrm{d}{x}\\ &= \int_{1.0}^{1.075}0.4\cdot 200(x-1) \mathrm{d}{x} + \int_{1.075}^{1.1} 0.6\cdot 10 \mathrm{d}{x}\\ &=80\cdot[x^2/2-x]_{1.0}^{1.075} + 6\cdot[x]_{1.075}^{1.1}\\ &=0.225 + 0.15\\ &=0.375 \end{align*}$$
Our model assumes that both distributions have the same covariance matrix. We get this in practice by taking the weighted average of the two:
As usual, once the model has been specified, the rest (inference for parameters and application) can be executed through straight probability theory.
Preliminaries
Goal
Introduction to linear generative classification with a Gaussian-categorical generative model
Materials
Mandatory
These lecture notes
Optional
Bishop PRML book (2006), pp. 196-202 (section 4.2 focuses on binary classification, whereas in these lecture notes we describe generative classification for multiple classes).
Application: Class prediction for new Data
the posterior class probability
Let's apply the trained model to predict the class for a "new" input $x_\bullet$:
$$\begin{align*} p(\mathcal{C}_k|x_\bullet,D ) &= \int p(\mathcal{C}_k|x_\bullet,\theta ) p(\theta|D) \mathrm{d}\theta \\ &= \sigma\left( \beta_k^T x_\bullet + \gamma_k\right) \tag{4} \end{align*}$$
where
$$\sigma(a)_k \triangleq \frac{\exp(a_k)}{\sum_{k^\prime}\exp(a_{k^\prime})}$$
is called a softmax (a.k.a., normalized exponential) function, and
$$\begin{align*} \beta_k &= \hat{\Sigma}^{-1} \hat{\mu}_k \\ \gamma_k &= - \frac{1}{2} \hat{\mu}_k^T \hat{\Sigma}^{-1} \hat{\mu}_k + \log \hat{\pi}_k \end{align*}$$
Bayesian Generative Classification
Markers
Code
Note that the binary class selection variable $y_{nk}$ groups data from the same class.
Fanta or Orangina? (**)
You have a machine that measures property $x$, the "orangeness" of liquids. You wish to discriminate between $C_1 = \text{`Fanta'}$ and $C_2 = \text{`Orangina'}$. It is known that
$$\begin{align*} p(x|C_1) &= \begin{cases} 10 & 1.0 \leq x \leq 1.1\\ 0 & \text{otherwise} \end{cases}\\ p(x|C_2) &= \begin{cases} 200(x - 1) & 1.0 \leq x \leq 1.1\\ 0 & \text{otherwise} \end{cases} \end{align*}$$
The prior probabilities $p(C_1) = 0.6$ and $p(C_2) = 0.4$ are also known from experience.
(a) A "Bayes Classifier" is given by
$$ \text{Decision} = \begin{cases} C_1 & \text{if } p(C_1|x)>p(C_2|x) \\ C_2 & \text{otherwise} \end{cases}$$
Derive the optimal Bayes classifier.
(b) The probability of making the wrong decision, given $x$, is
$$p(\text{error}|x)= p(C_1|\text{we-decide-} C_2, x) + p(C_2|\text{we-decide-}C_1, x)$$
Compute the total error probability $p(\text{error})$ for the Bayes classifier in this example.
Why Be Bayesian?
A student in one of the previous years posed the following question at Piazza:
"After re-reading topics regarding generative classification, this question popped into my mind: Besides the sole purpose of the lecture, which is getting to know the concepts of generative classification and how to implement them, are there any advantages of using this instead of using deep neural nets (DNN), as they seem simpler and more powerful?"
The following answer was provided:
If you are only interested in approximating a function, and you have lots of examples of desired behavior, then often a non-probabilistic DNN is a fine approach. However, if you are willing to formulate your models in a probabilistic framework, you can frequently improve on the deterministic approach in many ways. We list a few below:
1. Bayesian Evidence as a Performance Metric
Model performance is evaluated using the evidence $p(D|m)$ for a model, which inherently balances fit and complexity. This enables the use of the entire dataset for learning: there is no need for arbitrary splits into training and test sets.
2. Parameter Uncertainty Enables Active Learning
By maintaining uncertainty over model parameters, Bayesian models support active learning, i.e., the selection of data points that are expected to be most informative (See the lesson on intelligent agents). This allows learning from smaller datasets, unlike deterministic deep networks, which often require massive amounts of labeled data.
3. Predictions with Confidence Bounds
Bayesian models naturally yield predictive distributions, enabling uncertainty quantification (e.g., confidence intervals) around predictions.
4. Explicit and Modular Assumptions
Priors, likelihoods, and structural assumptions are explicitly specified and can be independently modified, promoting transparency and model modularity.
5. Unified Treatment of Accuracy and Complexity
Both data fit and model complexity are scored in the same probabilistic units. In contrast, how would you penalize overparameterized architectures (e.g., deep networks) in a deterministic framework?
6. Data-Dependent, Optimal Learning Rates
Learning rates emerge naturally from Bayesian updates. Contrast this with the trial-and-error tuning needed in standard optimization.
Example: The Kalman gain is an optimal learning rate based on current uncertainty.
7. Principled Knowledge Transfer
Bayesian inference enables posterior-to-prior propagation: results from one experiment (posterior) can inform the next (as a prior). This provides a principled mechanism for sequential learning and integration of heterogeneous information sources.
Admittedly, the probabilistic approach can be challenging to grasp at first, but the effort often pays off. It provides a principled, flexible, and robust framework for reasoning under uncertainty.
Challenge Revisited: Apple or Peach
We now have class-conditional Gaussian distributions for each class:
The softmax function
The softmax function can be viewed as a smooth approximation to the maximum function. Importantly, we did not impose the softmax posterior by assumption; rather, it emerged naturally by applying Bayes rule to our chosen prior and likelihood models.
Closing Thoughts
🎯 Key concept
ML estimation for $\{\pi_k,\mu_k,\Sigma\}$ in the GCM model breaks down to simple density estimation for Gaussian and multinomial/categorical distributions.
Estimate class-conditional distributions
Now, for each class, we fit a Gaussian distribution to the data of that class:
Therefore, the contours of equal probability (also known as discriminant functions, or decision boundaries), given by
$$\log \frac{{p(\mathcal{C}_k|x,D )}}{{p(\mathcal{C}_j|x,D )}} \overset{!}{=} 0 \,,$$
are lines (hyperplanes) in the feature space.
We'll apply the above results to solve the "apple or peach " example problem.
Implementation Issues
We can fit a Bernouilly distribution to y (see definition), this is simple:
Discrimination Boundaries
The class log-posterior $\log p(\mathcal{C}_k|x) \propto \beta_k^T x + \gamma_k$ is a linear function of the input features.
Click for the solution
$$\begin{align} &\log \frac{{p(\mathcal{C}_k|x,D )}}{{p(\mathcal{C}_j|x,D )}} \overset{!}{=} 0 \\ \implies &\frac{\beta_k^T x + \gamma_k}{\beta_j^T x + \gamma_j} = 1 \\ \implies &\beta_k^T x + \gamma_k = \beta_j^T x + \gamma_j \\ \implies &\beta_{kj}^T x + \gamma_{kj} = 0 \quad \text{(this is a line)}\,, \end{align}$$
where we defined $\beta_{kj} \triangleq \beta_k - \beta_j$ and similarly for $\gamma_{kj}$.
Click for the solution
Substitute 4.64 into 4.58 to get
$$\begin{align*} a &= \log \left( \frac{ \frac{1}{(2\pi)^{D/2}} \cdot \frac{1}{|\Sigma|^{1/2}} \cdot \exp\left( -\frac{1}{2}(x-\mu_1)^T \Sigma^{-1} (x-\mu_1)\right) \cdot p(C_1)}{\frac{1}{(2\pi)^{D/2}} \cdot \frac{1}{|\Sigma|^{1/2}}\cdot \exp\left( -\frac{1}{2}(x-\mu_2)^T \Sigma^{-1} (x-\mu_2)\right) \cdot p(C_2)}\right) \\ &= \log \left( \exp\left(-\frac{1}{2}(x-\mu_1)^T \Sigma^{-1} (x-\mu_1) + \frac{1}{2}(x-\mu_2)^T \Sigma^{-1} (x-\mu_2) \right) \right) + \log \frac{p(C_1)}{p(C_2)} \\ &\qquad\vdots \\ &=( \mu_1-\mu_2)^T\Sigma^{-1}x - 0.5\left(\mu_1^T\Sigma^{-1}\mu_1 - \mu_2^T\Sigma^{-1} \mu_2\right)+ \log \frac{p(C_1)}{p(C_2)} \end{align*}$$
Substituting this into the right-most form of (4.57) we obtain (4.65), with $w$ and $w0$ given by (4.66) and (4.67), respectively.
Making a Decision
How should we classify a new input $x_\bullet$?
The Bayesian answer is to compute the posterior distribution over classes,
$$p(\mathcal{C}_k | x_\bullet,D)\,,$$
and this completes the classification task: the posterior encapsulates all available information about class membership given the input and data.
If a definite classification must be made, a natural choice is the class with the highest posterior probability:
$$\begin{align*} k^* &= \arg\max_k p(\mathcal{C}_k|x_\bullet,D) \\ &= \arg\max_k \left( \beta _k^T x_\bullet + \gamma_k \right) \end{align*}$$
This corresponds to a maximum a posteriori (MAP) decision rule, which is both simple and effective in many practical settings.
Posterior class probability of $x_∙$ (prediction)
Now we can answer the question from the challenge, and calculate the probability that $x_∙$ is an apple .
Click for the solution
Following the same derivation as above (in the cell "Click for proof of (4)"), the posterior class probability evaluates to
$$\begin{align*} p(\mathcal{C}_k|x_\bullet,D ) \propto \exp \Big\{ \underbrace{-\frac{1}{2}x_\bullet^T \hat{\Sigma}_k^{ - 1} x_\bullet}_{\text{now a function of }k} + \underbrace{\hat{\mu}_k^T \hat{\Sigma}_k^{-1}}_{\beta_k^T} x_\bullet \underbrace{- {\frac{1}{2}}\hat{\mu}_k^T \hat{\Sigma}_k^{ - 1} \hat{\mu}_k + \log \hat{\pi}_k }_{\gamma_k} \Big\} \,. \end{align*}$$
Because the quadratic term $x_\bullet^T \hat{\Sigma}_k^{-1} x_\bullet$ is now class-dependent, the decision boundaries are given by quadratic equations in $x$. Hence, the decision boundaries are generally (hyper-)ellipsoid surfaces.
🎯 Key concept
If the class-conditional distributions are Gaussian with equal covariance matrices across classes (i.e., $\Sigma_k = \Sigma$), then the discriminant functions are hyperplanes in feature space.
You are also given a test fruit , which has known feature values but an unknown fruit label.
Now assume that the class-conditional feature distributions are modeled with class-dependent covariance matrices, i.e.,
$$ p(x_n|\mathcal{C}_{k}) = \mathcal{N}(x_n|\mu_k,\Sigma_k) $$
What do the decision boundaries look like in this case?
Bishop exercise 4.8 (**)
Using (4.57) and (4.58) (from Bishop's book), derive the result (4.65) for the posterior class probability in the two-class generative model with Gaussian densities, and verify the results (4.66) and (4.67) for the parameters $w$ and $w0$.
Model Specification
Representation
The above data set will be represented as $D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}$, where
inputs $x_n \in \mathbb{R}^M$ are called features.
outputs $y_n \in \mathcal{C}_k$, with $k=1,\ldots,K$; The discrete targets $\mathcal{C}_k$ are called classes.
Similar to our representation of the categorical distribution, we will again use the 1-of-K (or one-hot) encoding to represent the discrete class labels. We define binary class selection variables
$$y_{nk} = \begin{cases} 1 & \text{if } \, y_n \in \mathcal{C}_k\\ 0 & \text{otherwise} \end{cases}$$
Hence, the notations $y_{nk}=1$ and $y_n \in \mathcal{C}_k$ mean the same thing.
Problem
Based on the observed data, what is the probability that the test fruit is an apple , and not a peach ?
Solution
The plan for generative classification is as follows: We begin by constructing a model for the joint distribution
$$p(x, y) = p(x | y)\, p(y),$$
which combines a class-conditional likelihood $p(x | y)$ with a prior $p(y)$ over classes. Then, we apply Bayes rule to compute the posterior class probabilities,
$$p(y|x) = \frac{p(x|y) p(y)}{\sum_{y^\prime} p(x|y^\prime) p(y^\prime)} \propto p(x|y)\,p(y)$$
This posterior can then be used for classification by selecting the class with the highest probability.
Next, we discuss the three modeling stages: (1) model specification, (2) parameter learning, (3) application (classification).
We will refer to this model (specified by Eqs. 1 and 2) as the Gaussian-Categorical Model (GCM).
N.B. In the literature, this model (with possibly unequal $\Sigma_k$ across classes) is often called the Gaussian Discriminant Analysis model and the special case with equal covariance matrices $\Sigma_k=\Sigma$ is also called Linear Discriminant Analysis. We think these names are a bit unfortunate as it may lead to confusion with the discriminative method for classification.
Prior
We use a categorical distribution for the class labels $y_{nk}$:
$$p(\mathcal{C}_{k}) = \pi_k \tag{2}$$
We can use this function on every point of the plot, to draw a heatmap of apple probability:
\(p(\text{apple}|x=x_∙) = 0.679\)
persist_js_state$876f47d8-b272-4e23-b5ec-5c7d615ff618running§runtime)!depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$876f47d8-b272-4e23-b5ec-5c7d615ff618depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body persist_js_state$f7a19975-a919-4659-9b6a-d8963a1cd6d9running§runtime9depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$f7a19975-a919-4659-9b6a-d8963a1cd6d9depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&W has_pluto_hook_features¤bodyChallenge: Apple or Peach?
Maximization of log-likelihood
Maximization of the LLH for the GDA model breaks down into
Gaussian density estimation for parameters $\mu_k, \Sigma$, since the first term contains exactly the log-likelihood for MVG density estimation. We've already done this; see the Density Estimation lesson.
Multinomial density estimation for class priors $\pi_k$, since the second term holds exactly the log-likelihood for multinomial density estimation. We've also done this before; see the Density Estimation lesson.
Exercise: Discrimination boundaries
Loading...
Show that the log-likelihood for the parameters evaluates to
$$\log\, p(D|\theta) = \sum_{n,k} y_{nk} \underbrace{ \log\mathcal{N}(x_n|\mu_k,\Sigma) }_{ \text{see Gaussian lecture} } + \underbrace{ \sum_k m_k \log \pi_k }_{ \text{see multinomial lecture} } \tag{3}$$
where we used $m_k \triangleq \sum_n y_{nk}$.
Click for proof of (4)
$$\begin{align*} p(\mathcal{C}_k|x_\bullet,D ) &= \int p(\mathcal{C}_k|x_\bullet,\theta ) \underbrace{p(\theta|D)}_{=\delta(\theta - \hat{\theta})} \mathrm{d}\theta \\ &= p(\mathcal{C}_k|x_\bullet,\hat{\theta} ) \\ &\propto p(\mathcal{C}_k)\,p(x_\bullet|\mathcal{C}_k) \\ &= \hat{\pi}_k \cdot \mathcal{N}(x_\bullet | \hat{\mu}_k, \hat{\Sigma}) \\ &\propto \hat{\pi}_k \exp \left\{ { - {\frac{1}{2}}(x_\bullet - \hat{\mu}_k )^T \hat{\Sigma}^{ - 1} (x_\bullet - \hat{\mu}_k )} \right\}\\ &=\exp \Big\{ \underbrace{-\frac{1}{2}x_\bullet^T \hat{\Sigma}^{ - 1} x_\bullet}_{\text{not a function of }k} + \underbrace{\hat{\mu}_k^T \hat{\Sigma}^{-1}}_{\beta_k^T} x_\bullet \underbrace{- {\frac{1}{2}}\hat{\mu}_k^T \hat{\Sigma}^{ - 1} \hat{\mu}_k + \log \hat{\pi}_k }_{\gamma_k} \Big\} \\ &\propto \frac{1}{Z}\exp\{\beta_k^T x_\bullet + \gamma_k\} \\ &= \sigma\left( \beta_k^T x_\bullet + \gamma_k\right) \end{align*}$$
You're given the numerical values for two features (let's say, sugar content and acidity) of a bunch of fruits. Each piece of fruit is either an apple or a peach .
Generate this data yourself by selecting the total number of fruits with the slider:
Click for the solution
$$\begin{align*} \log\, p(D|\theta) &= \log \prod_n p(x_n,y_n|\theta ) \quad \text{(assume IID data)} \\ &= \sum_n \log p(x_n,y_n|\theta ) \\ &= \sum_n \log \prod_k p(x_n,y_{nk}=1\,|\,\theta)^{y_{nk}} \\ &= \sum_{n,k} y_{nk} \log p(x_n,y_{nk}=1\,|\,\theta) \\ &= \sum_{n,k} y_{nk} \log p(x_n|y_{nk}=1) + \sum_{n,k} y_{nk} \log p(y_{nk}=1) \\ &= \sum_{n,k} y_{nk} \log\mathcal{N}(x_n|\mu_k,\Sigma) + \sum_{n,k} y_{nk} \log \pi_k \\ &= \sum_{n,k} y_{nk} \log\mathcal{N}(x_n|\mu_k,\Sigma)+ \sum_k m_k \log \pi_k \end{align*}$$
Our data is structured as follows:
$y$ is whether a data point is an apple (
true) or a peach (false)$x$ has one column of data per fruit